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ABSTRACT 

The results of three years of evaluative 
investigation of the University of Pittsburgh Learning Research and 
Development Center's program of individualized education are 
described. The study was conducted in seven Follow Through sites and 
three Pittsburgh area schools. Standardized tests were used as input 
and outcome measures; questionnaires and videotapes were used to 
gather information about the classroom processes. The results 
indicate that over the three-year period the implementation of the 
program continued to move toward ideal goals. Extensive data 
reduction procedures were used and the rationale of usage discussed. 
Results also point to input as the primary explanation of student 
end-of-year performance, though classroom processes contribute a 
small but consistent amount. The data indicate greater ease in 
identifying negative factors in successful classroom processes than 
in determining the positive factors. (Author/HV) 
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Abstract 



This study describes the results of three years of evaluative investigation 
of LRDC's program of individualized education. Hie study was conducted 
in seven Follow Through sites and three Pittsburgh area schools. Standard- 
ized tests were used as input and outcome measures; questionnaires and 
videotapes were used to gather information about the classroom processes. 
The results indicate that over the three-year period the implementation of 
the program continued to move toward ideal goals. Extensive data reduc- 
tion procedures were used and the rationale of usage discussed. Results 
also point to input as the primary explanation of student end-of-year per- 
formance, though classroom processes contribute a small but consistent 
amount. The data indicate greater ease in identifying negative factors in 
successful classroom processes than in determining the positive factors. 
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PROGRAM EVALUATION: 
AN EMPIRICAL STUDY OF INDIVIDUALIZED INSTRUCTION 



Gaea Leinhardt 

Learning Research and Development Center 
University of Pittsburgh 

The area of program evaluation arises from the need to assess 
whether or not a specific policy or program has altered a trend or pattern 
of events in the desired way. The policy or program may be one designed 
to reduce or eliminate something such as crime or disease, or it may be 
designed to increase or generate something such as employment, educa- 
taional attainment, or the number of new minority businesses. In address- 
ing the general problem of determining the effect of a particular policy or 
program, the specific problem of assessing the actual presence of a pro- 
gram in the field must be dealt with. To state it in a different way, the 
task is to isstablish a reasonable line of causality showing that a change 
occurred due to the presence of an implemented program. Assessing 
implementation is the focus of this paper. 

Compensatory education is one example of a publicly funded program 
whose effectiveness requires empirical evaluation efforts. The idea of 
compensatory education is to compensate specific groups of children for 
an initial educational deficit by providing and funding special programs 
for them. Among the educational programs established in this way were 
Head Start and Follow Through. Head Start, as its name suggests, was 
designed to give economically disadvantaged children an early educational 
start at 3, 4, and 5 years of age. Follow Through was designed to protect 
and extend the presumed gains made in Head Start through the third grade 



of public education. It should be realized that these compensatory educa- 
tional programs are not unitary. There is. for example, no single FxdIIow 
Through program; there are at least 22 identifiable programs, and within 
the 22 there are probably several variations of sufficient magnitude to be 
considered as separate programs. 

A separate federal program, the Cooperative Research Act of 1954, 
led to the establishment of over 40 research and development centers and 
educational laboratories. The research and development (R&D) centers 
were, for the most part, university-based organizations with specific 
"missions" to improve the quality of education in the United States. The 
educational laboratories, on the other hand, were established to refine 
and disseminate the findings and products of the R&D organizations to the 
educational community of the United States. 

This paper presents empirical results from program evaluation 
undertaken at one of these R&D organizations, the Learning Research and 
Development Center (LRDC) at the University of Pittsburgh. The study 
is directed at evaluating LRDC's individualized instructional program in 
the context of the Follow Through system. Three concerns are addressed 
(a) changes in levels of implementation of the major components of the 
program; (b) ways in which large amounts of intercorrelated data on class 
room practices can be reduced and analyzed; and (c) the change or lack of 
change in the relation between classroom processes and student achieve- 
ment. 

Theoretical framework. It is important to view the research re- 
ported here in the light of its theoretical and practical roots, which lie 
in several rather separate areas. First, the motivation for the initial 
study grew out of the practical need for evaluation of LRDC's educational 
program in the Follow Through schools. Second, the techniques used to 
provide that evaluation came from two sources: observational literature 
and natural field-based research. 
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Most major models of program evaluation recognize the need for 
process information in order to interpret outcome results (Lindvall & Cox, 
1970). Evaluative research, which aims to identify outcomes as well as 
explain the cause of outcomes, requires a reasonably detailed account of 
what the actual treatment was rather than whcxt it was supposed to be 
(Charters h Jones, 1973). An additional requirement is that the treat- 
ment must be examined in an environment secure from extensive developer 
influence (Scriven, 1973). 

In spite of the fact that the theoretical evaluatipji literature recog- 
nizes the need for process information, that literature itself is of little 
help in providing practical advice on its collection. This information has 
been gleaned instead from writers in the area of field-based research (e.g., 
Campbell, 1969), by examples from other evaluation researchers (e.g., 
Keeves, 1972; Stallings, Note 1), and from researchers in the area of 
classroom observation (e. g. , Rosenshine, 1970). 

Setting 

The research was carried out during the 1973-74 academic year. 
It is a continuation and follow-up of research conducted during 1971-72 
and 1972-73 (Leinhardt, 1974b). The evaluative effort has been focused 
at the second-grade level across several sites^ rather than across grade 
levels at one or two sites on the assumption that variance is maximized 
within a grade level at different sites and minimized within a site- 



^Site: A portion or all of a school district which has chosen an early 
childhood education program for use in at least one grade level in at least 
one school. In this paper, a site refers to at least one school using the 
LRDC model in a district. A local site is a school in the Pittsburgh area 
using the CRDC program without Follow Through funding. 
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Background , LRDC is one of 22 national FoUov/ Through sponsors,^ 
Each sponsor in the Follow Through system is responsible for inriplement- 
ing its own program at those sites which have chosen that particular model. 
LRDC implements its reading and mathematics programs in seven Follow 
Through sices and three local sites (for a brief program description, see 
Appendix A)^ This means that the programs are implemented to some 
degree in 61 second-grade classrooms. The LRDC Follow Through sites 
extend from North Dakota to Arkansas to West Virginia. The ten sites are 
located in urban, rural, and suburban environments; their student popula- 
tions range from predominately white to predominately black, and one site 
is predominately native American Indian. 

History , From 1969 to 1971, LRDC kept informal and often undocu- 
mented records of the implementation of its programs at the Follow Through 
sites. This effort was essentially an inventory of the presence or absence 
of vital materials or practices in the classrooms. Toward the end of 1970, 
it became clear that it would be useful to document program implementa- 
tion on an expanded basis. Work was begun on defining basic dimensions 
of the program and constructing measures for each dimension (Leinhardt, 
1972). 

From 1971 to 1974, the evaluation project within LRDC^ monitored 
the basic classroom processes that occurred in all second-grade class- 
rooms in which some aspect of the program was implemented. The pur- 
pose of this monitoring effort was to assess the degree to which features 



^ Sponsor ; An independent organization responsible for helping a site 
to implement an early childhood education program developed by the spon- 
sor. 

^LRDC is an organization of approximately 250 employees, 35 of whom 
ore at the Ph. D level. The center is organized into 25 projeqjis, each with 
one or more Ph. Ds and a support staff. 
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of LRDC's program were actually implemented in cla^^srooms. The con- 
cern has changed, then, from a focus on an inventory to a focus on the 
degree to which the spirit of the program has been transplanted. This lat- 
ter emphasis has itself gone through some changes and is moving toward 
the collection of information about classroom processes, independent of 
particular programs, that affect educational outcomes (Leinhardt, 1974b; 
Cooley & Leinhardt, 1975). 

Procedure 

During the 1973-74 school year, student performance and classroom 
process data were collected. Ihese data were collected at seven Follow 
Through sites and three local schools, all of which were using LRDC cur- 
ricula in math and reading. These data came from four sources: standard- 
ized tests, questionnaires, videotapes, and student records* 

Aptitude and achievement data were obtained from three sets of 
standardized tests. The Lorge -Hagen- Thorndike Test of Cognitive Abili- 
ties (Thorndike, Hagen, & Lorge, 1968) was administered in the fall and 
served as an estimate of initial student ability. The Raven Coloured Pro- 
gressive Matrices (1956) was administered in the spring and was one of 
three measures used to estimate student abilities at the end of the year. 
The Metropolitan Achievement Test (MAT) Form F or G (Durost, Bixler, 
Wrightston, prescott, & Balow, 1971) was also administered in the spring 
and was used to estimate achievement in reading and mathematics (total 
Gubtest scores were used). These tests were then aggregated at the class- 
room level to estimate average class performance on the four standard- 
ized measures. 

Data on classroom practices were obtained from responses to a 
questionnaire designed to yield information on a variety of topics. This 
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questionnaire was completed by the educational specialist"* at each site 
with the help, where appropriate, of the classroom teacher. (The ques- 
tionnaire is reproduced in Appendix B. ) The questionnaire was completed 
twice, once in the fall and once in the spring. The two responses were 
averaged, unless obvious errors were detected. This questionnaire gen- 
erated data on 33 variables, of which 18 were retained for further analysis. 

Data on the teaching practices in each classroom were obtained from 
videotapes. Each classroom was videotaped during reading and mathe- 
matics In the fall and spring. The tapes were made by the educational 
specialists and coded by the evaluation group at LRDC. Sixty minutes of 
videotape were recorded for each classroom. The analysis of these tapes 
yielded information on 30 variables, of which 20 were retained for further 
analysis. 

Data on student performance withiT^ the curricula were obtained from 
the students' school records. Student initial and final location in both the 
reading and jna thematic s curricula were recorded and used in combination 
with the standardized test results." 

The reliability of the instruments listed above has been estimated 
from two points of view: (a) the stability of the characteristics being 
measured over the year's period, and (b) the intercoder reliability. The 
stability of the variables measured by the questionnaire has been estimated 
at between . 76 and . 90 (Leinhardt. 1974b), and stability or generalizability 
of the variables measured by the videotapes has been estimated at .58. 
Intercoder reliability for the questionnaires and videotapes was estimated 
at . 98 and . 95, respectively. 



'♦Educa tional Specialist; Individuals ) responsible at the site for 
implementing the sponsor's program(s). 
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Findings 

In reporting the findings of this study, three questions are addressed. 
First, how has the implementation of a program of education changed over 
the course of three years? Second, how can. the large mass of data gen- 
erated hy this investigation be reduced in a way which maintains the mean- 
ing but reduces the confusion of dealing with large numbers of variables 
and classrooms? Third, what is the impact of classroom processes on 
student achievement, how should these processes be considered, and how 
have they changed over the three-year period? 

Implementation 

In this section, the changes in classroom processes and the data 
which reflect them are examined. Before reviewing the details of the 
available data, some general comments are provided to serve as an over- 
view* 

From the 1971-72 school year to the 1973-74 school year, in accord- 
ance with the federal guidelines for Follow Through, the number of spon- 
sor service days to the sites decreased. This is indicative of an initial 
policy decision which aimed at installing a program and then weaning the 
sites and sponsors from each other. Thus, the sites were expected to 
require and receive considerable aid initially in implementing the pro- 
gra;m, but they were expected to require and receive fev.er and fewer sup- 
ports as time passed. One would expect that if the program were difficult 
to implement or merely poorly implemented, there would be an increase 
over time of activities or behaviors that depart strongly from the sponsor's 
model. One would also expect that given a pattern of reduced support over 
time, teachers and specialists would begin to introduce their own innova- 
tions in answer to problems that arise. The question is whether or not 
those ''answers** are in harmony with the spirit of the sponsor's model. 
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As the data on implementation are reviewed, two questions should 
be kept in mind: (a) Are key features of individualization maintained in 
the face of a reduction of support? (b) Does implementation increase 
over time in the face of increased experience with the program? Four- 
teen variables are presented in Table 1. These variables were selected 
from the many gathered each year when there were at least two years of 
data available on them. Table 1 shows the range of site means for the 
four sites on which we have been consistently gathering data- Each site 
has between four and eight second-grade classes. 



Table 1 

Mean Ranges for Classroom Process Information 
over Three Years with Original Four Sites 



Variable 


197M972 


1972-1973 


1973-1974 


Teacher experience with LRDC program 


.00 


1.83 


.70 


- 2.20 


1.50 - 


3.20 


Class size 


21.00 


27.40 


20.60 


- 25.30 


20.50 


25.80 


Percentage of students attending 


87.80 - 


96.60 


91.00 


- 96.70 


94.20 - 


95.80 


Percentage of unique assignments 


51.00 " 


57,00 


77.00 


- 86.20 


80.00 - 


85.50 


Percentage of blocked assignments 






47.30 


- 63.80 


40.00- 


73.50 


Number of days since the last test 


3.15- 


9.06 


3.40 


- 4.50 


2.60 - 


4.20 


Minutes per day in mathematics 


44.16- 


60.00 


44.20 


- 60.00 


60.20 • 


70.70 


Minutes per day in reading 


8C.30 - 


120.00 


76.00- 


-110.20 


81.20 - 


98.30 


Mathematics placement 


6.35 - 


18.90 


7.00 


- 20.90 


9.80 - 


19.20 


Reading placement 


1.82 - 


4.87 


3.40 


- 13.10 


3.00 • 


9.20 


Mathematics progress 


6.26 • 


11.56 


7.90 


- 9.50 


9.00 - 


11.00 


Reading progress 


6.60 - 


6.86 


5.00 


7.90 


6.00 


7.70 


Number of adults who travel 


1.50 - 


2.30 






1.50 - 


2.E0 


Percentage of negative statements 


1.00- 


6.00 






.80 


4.00 


Ratio of mean cognitive contacts to 
mean management contacts 


.99 








1.39 





8 

12 



These variables are designed to provide information for the evalua- 
tive analysis. Since they are specific to this effort and are not generally 
known, they require some discussion as to meaning and utility. Rather 
than describing each variable in the table, a few of the more important 
ones will be reviewed. Attendance is often considered an indicator of 
student attitude toward school; high attendance indicates a more positive 
attitude and low attendance indicates negative attitudes. Regardless of 
whether or not that interpretation is warranted, attendance is important 
because it is unlikely that a student can learn much from school without 
being there. It is, therefore, encouraging to see that the lower end of the 
range of percentage of students attending has gone up from 87.8% to 94. 2% 
attending. 

Three variables- -unique assignments, blocked assignments, and 
frequency of testing- -relate to teacher assignment practices that are 
specified by the instructional model. One of these, unique assignments, 
will be discussed in detail. Individualized instruction requires that each 
student receive only instruction that he or she needs. It is assumed that 
if ail students in a class simultaneously receive the same instruction, 
individual students will not be receiving exactly what they need. The per- 
centage of unique assignments estimates how unique the entire group of 
class assigiiments are; the more unique, the higher the percentage. 
Clearly, it is not necessary for every child to receive a different assign- 
ment; but, an increase over time from approximately 54% to 83% unique 
is indicative of increasing individualization. 

Four variables - -math and reading placement and progress- -relate 
to the initial starting position of children in the curriculum and their 
progress during the year. These variables are indicative of the impact 
of the instructional practices of not only the second grade, but also of 
the first grade. Both mathematics and reading placement have gone up 
over time. This is impressive because basic entering ability has remained 



stable, atfd higher placement levels thus indicate that more material is 
being taught in the first grade and retained in entering second grade. The 
students have made roughly similar progress over the three years in both 
subject areas, indicating that more material had been taught and, presuma- 
bly, learned at the end of the second grade in 1974 than in 1972. 

Several variables relate to teaching practices in the 

classroom. One of these, the i ^oi e to management contacts, 

is indicative of the amount of emphatiis placed by the teacher on instruc- 
tional rather than managerial information. The larger the number, the 
more the teacher is interacting in an instructional way with students. 
Hiis number has increased slightly over time. The LRDC instructional 
model seeks to minimize the dependence of children on teachers as mana- 
^ gers, dispensers, and clerks. Its implementation should lead to a reduc- 
tion in the number of managerial comments made, and the data support 
this contention. 

Other variables in the table support a claim that the level of imple- 
mentation of the LRDC program has remained stable during the three 
years in which support- -in number of sponsor service days- -was gradually 
* being withdrawn. The number of adults in the room who ^'travel" (or circu- 
late) during the working period has remained stable and higher than is 
indicated by the model. The percentage of negative comments made during 
an observed period of time has remained quite low and stable over the 
three years. The amount of time spent in teaching reading and mathematics 
has fluctuated, but is within developer guidelines, and testes are adminis- 
tered regularly in order to prescribe assignments effectively. 

There is much, however, that is not revealed by these data. Over 
the three years that we. have been observing teachers in classrooms using 
our programs, there has been a steady increase in the confidence with 
which the model has been implemented. The teachers and aides travel in 
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the room, helping, tutoring, and quizzing students in order to diagnose 
their instructional needs. Specific program sequences are frequently 
altered on the basis of the accumulated wisdom of teachers and educational 
specialists. Different patterns of classroom organization are constantly 
being tried. Some teachers travel among the students as they do their pre- 
scribed work while the aide tests or engages in extended tutoring.- In other 
situations, both the teacher and the aide travel among the students for a 
while to get them started on their separate 1 i id then engage in rela- 
tively brief , small group instruction. In stiT ^Iher cases, the teacher 
starts with small group instruction and follows that by joining the aide in 
traveling. In many cases, teachers have students help one another. Com- 
mon to all of these patterns of interaction, however, is an attempt to imple- 
ment the idea of individualized instruction, ^'^©ver time, these four sites 
have increasingly captured the spirit of individualization. In those areas 
which are critical to the implementation of the instructional model, the 
sites have maintained or improved their practices. In those areas in which 
precise implementation is not as critical, the sites have expressed their 
own uniqueness. 

Data Reduction 

In the process of assessing the implementation of the instructional 
program, an enormous quantity of inter correlated data is generated. Over 
the last three years, a haunting problem has been the reduction of the large 
number of process measures for purposes of a more complete analysis. 
There are two reasons why it is necessary to reduce the mass of classroom 
process data. First, there is the obvious problem of limited degrees of 
freedom for any analysis which requires that the number of variables be 
reduced by approximately a factor of 10. The second need for data reduc- 
tion arises from the problems of trying to understand and interpret what 
all of the numbers mean. The trade-off is between large numbers of 
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specific, concrete, easily understood bits of information and a total sys- 
tem of information. Although what is needed is to move toward a system 
of information, it often appear^ that detailed and specific information is 
of greater value for the practitioner. This problem has plagued most 
classroom researchers. It is common in almost any situation in which 
one attempts to relate detailed descriptive information to outcomes and in 
which one uses the classroom as the unit of analysis (e.g., see Stallings, 
Note 1). 

One way to solve ^ ' n of degrees of freedom is to change 
the unit of analysis fron ssroom to the student. It could easily be 

argued that in an individualized program the individual student is a more 
appropriate unit of analysis (e.g., see Brophy, Note 2), However, there 
are two reasons why it is inappropriate to consider the student as the unit 
in this case . First, the classrooms are self-contained, and although dif- 
ferent students receive different ^'treatments, " the students in a single 
class can hardly be said to vary independently of one another with respect 
to that treatment. Second, stable and reliable information on the indi- 
vidual student's sets of experiences is very costly to collect, requiring 
a minimum of two to two and one -half days of observations per child . 
Thus, using the classroom as the unit of analysis has permitted us to 
collect sufficient information to estimate the stability of procedures, 
the reliability of coding, and so on (Leinhardt, 1974a). 

Data reduction: Possible approaches . Basically the problem of re- 
ducing data is the following: How can the number of variables be decrease 
without losing potentially valuable r-iformation? Clearly, the first step is 
to eliminate from consideration measures which cannot provide informa- 
tion for further analysis. This includes measures for which there is no 
difference among classrooms or for which there are obvious inaccuracies 
in the data reported. It should be remembered, however, that no vari- 
ance in a measure does not mean that the measure has no information. 
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For example, if one finds that every child in every classroom has a mathe- 
matics textbook, that is important information. However, that particular 
variable in that particular sample will not be useful in predicting student 
achievement because there is no variance. After measures with no vkrir;?:=- 
ance or with obvious errors are eliminated, the next step is to see if two 
or more of the measures are measuring the same thing. This information 
is obtained by examining correlation and partial correlation (with student 
entering abilities controlled) matrices. If two or more variables correlate 
highly (above .80) with each other and correlate to the same magnitude with 
the dependenf • ), it is possible that they are measuring similar 

phenomena. - > cau then be combined, or one of the set can be substituted 
for the others. These two procedures resulted in the elimination of 26 
measures in the current data set. 

One way of accomplishing the second step is to use some type of fac- 
tor analysis of the process data. This technique has been successfully 
employed by Soar and Spar (1972) and others. My experience, however, 
has been that factor patterns tend to be illogical, that is, the variables that 
go together mathematically do not do so in^tellectue and do not replicate 
very well from, year to year .Leinlxardt, 1972). In past, therefore, 
either the sperrific correlation or partial correlatio d single variables 
with dependerr measures has been reported (Leinha.rr 1974a), or the 
data have been combined using a different approach jinhardt, 1974b). 

Another approach to combining variables is to use a model of class- 
room praqess and fit variables to the constructs of the model post hoc. 
The Cooley-Lohnes model (Cooley & Lohnes, in press), an expansion of 
the Carroll model (Carroll, 19^3), has been especially useful for this 
purpose (see JLeinhardt, 1 974b; Stalling s. Note 1; Caxoley & Emrick, Note 
3). The basic approach in using the Cooley- Lohnes model is to select 
variables vmim can logically be assigned to each of the model's six con- 
structs anc ttfen to combine the variables within each construct. Combining 
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the variables is accomplished either by factoring within the construct and 
summing using the factor scores or by transforming the raw data to stand- 
ard scores and summing using equal weights. The latter approach was 
tried on the 1972-73 data set after the original data set had been reduced 
to_.six constructs, two of which related to student performance and four of 
which related to process. They were analyzed using commonality analysis 
(Mood, 197 1). While this approach is extremely helpful in simplifying a 
large and complex data set, it is not completely satisfying if the data are 
collected with some other organizational scheme in mind and then are fit 
to the model post hoc. Any data analysis model is most usefully employed 
if the data are initially collected with the model in mind (see Cooley & 
Leinhardt, 1975 ). 

^gitg^Xgd ^^^^o^' Current strategies . Currently, the most useful 
approach seems to be to organize the data around some set of logically 
conceived clusters --en^cdr: as the basic domains employed in developing the 
instruTnentation--?-nd ttiijs:ii to examine each cluster separately for purposes 
of data reduction. Table 2 identifies the clusters, the specific variables 
in each cluster, a^cd tis procedure(s) which were employed to combine the- 
measures. InitiaSEx,, ^UL of the measures were converted to standard scores 
(zero mean and mrirt variance). TTien each cluster of variables was analyzed 
using principal crmmosnCTl analysis with one or two factor solutions. These 
factors Were the ^ exaxniTOd and used as guides for combining the data. If 
the factor analyH:::^^ suggested that: the variables equally related to a factor, 
we summed the sc^ire:s without weighting them, ::inaicing certain that ^^avorite 
variables were re&isined. Equal weights were useii for combining the follow- 
ing clusters: Inpon, Traveling Frequency I and II, and Outcomes. If the 
factor analysis sai-:rs5ted that the variables were ciifferentially related to a 
factor, the weights of the variable on the factor were used to sum them. 
This was done for t3ae :::a»ortunity and Ratings of Instructional Behavior 
clusters, 
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Table 2 

Map of Total Data Reduction 



INPUT 



PROCESS 



Input (Summed Z scores, equal weighting) 

Lorge ThorndikQ Cognitive Abilities Test 
Initial placement mathematics 
Initial placement reading 

Process Set 1 

Ratings off Instructional Behavior (Summed Z scores, weights suggested by factor patterns) 
Active responses sought 
Modeling of response 
Focusing attention 
Contingent praise 
General praise 
Clear tutoring 
Class management 
Backward chains 
Refers to earlier information 
Solicits opinion of correctness 
Refers to earlier success 

Opportunity (Summed Z scores, weights suggested by factor patterns) 
The number of adults traveling in mathematics 
The number of adults traveling in reading 
Availability of trained alternate 
Percentage of children present 
Timeestimate 

Minutes a day spent using LRDC programs 
Mi!:utes a day in large group activities 

Traveling Frequency I (Summed Z scores, equal weighting) 
Reading cognitive statements 
fteding cognitive questions 
^rSaading tutorings 
^hild initiated 

Proportion of cognitive statements to total stetements 

Autonomy H (Summed Z scores, equal weighting) 

Children decide what unit of math to study 
Children request special tutoring 
Children begin work as soon as they comeJn 
Peer tutoring occurs 

Assignments 

Percentage of unique assignments 

Process Set 2 

Autonomy I (Summed Z scores, equal weighting) 
Cntidren get own assignments 
Children get own prescription ticket 
Children decide when to take a test 

Children move back and forth between work end free activity 
Traveling Frequency II (Summed Z scores, equal weighting) 
Percentage of negative statements during readir>g 
Percentage of negative statements during mathematics 
Number of errors made during rending 
Number of errors made during mathematics 



Testing 

Process Set 3 

Context 



Number of days since the last test 



Teacher exporiencii with the program 



OUTCOME 



Outcomes (Summed Z scores, equal weighting) 
* Ravens Progressive Matrices 
MAT reading total 
MAT math total 
End-of'Year reading placement 
End'of'Year mathematics placement 
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Following all of these steps succeeded in reducing the ratings of 
teacher behaviors to one variable and the frequencies of specific behaviors 
to two variables, to cite the most extreme cases. This improved the situa- 
tion by leaving 10 clusters on which to predict one; however, with approxi- 
mately 10% attrition in the orginal number of observations due to missing 
data, there were still too many variables for the available degrees of free- 
dom. 

A correlation was run on the 11 variables followed by a partial corre- 
lation with input partialled out to suggest further reductions. Tables and 4 
show the results of these analyses. After examining the correlations and par- 
tial correlations between the process variables and outcomes, the next step 
suggested by the data was to break fee process variables into three groups: 



Table; 3 

Intercorrelations Among Clusters for 1973-74 Data 





Input 


Traveling Frequency 
Ratings Oppartunitv 1 II 


Autonomy 
1 II 


Days Since 
Teaching Percentage Last 
Experience Unique Test Outcome: 


Input 


1.00 














Ratings 


.01 


1.00 












Opportunity 


.10 


.35 


1.00 










Traveling frequency 1 


.23 


.07 


- .08 


1.00 




1 




Traveling frequency It 


• .23 


• .41 


- .19 


• .20 


1.00 






Autonomy 1 


.13 


• .14 


.11 


• .06 


- .00 


1.00 




Autonomy 11 


- .13 


.35 


.23 


.12 


- .14 


.15 1.00 




Teaching experience 


.61 


- .08 


.02 


.00 


. .03 


.10 • .08 


1.00 


Percentage unique 


.09 


.24 


.08 


• .02 


- 


.05 .08 


.10 1.00 


Days since last t^ 


• .14 


.17 


- .21 


.13 


- .06 


.06 .09 


.04 - .18 1.00 


Outcomes 


.91 


.12 


.20 


.27 


• .32 


• .01 • .01 


.49 .16 • .29 1.00 



Note: r. = 55. 
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Table 4 

Partial Correlations Among Clusters for 1973-74 Data 



Traveling Frequency Autonomy Days Since 

Teaching Percentage Last 
Ratings Opportunity ! M I H Experience Unique Test Outcomes 



Ratinas 


1.00 


















Opportunity 


.35 


1.00 
















Traveling frequency 1 


.07 


- .10 


1.00 














Tra\/plinn frpniipnpv 11 


. 42 


- .17 


• .16 


1.00 












Autonomy 1 


« .14 


.10 


• .09 


.03 


1.00 










Autonomy II 


.35 


.25 


.16 


- .18 


.17 


1.00 








Teaching experience 


' .11 


- .05 


- -.13: 


.14 


.03 


. .00 


1.00 






Percentage unique 


.24 


^07 


- J4 


- .31 


- .07 


.09 


.06 


1.00 




Days since last test- 


.17 


- .20 


.16 


- .10 


.08 


.07 


.15 


- .17 


1.00 


Outcomes 


.28 


.27 


.:6 


- .28 


- .30 


.26 


• .20 


.18 


- .40 1.00 



Note: n = 55. 



those which related positively to cnitcomes, those which related negatively 
to outcomes, and those which swirched in their relationship to outcomes 
when input was partialled out. The autonomy variables, which are uncor- 
related with outcomes, were assigned to one of the three groups based on 
the partial correlations. This organization of the data is also presented in 
Table 2; Process Se± 1 consists of those five variables which relate posi- 
tively to achievement, Process Set 2 consists of the three variables which 
relate oaegatively, and Process Set 3 consists of the single variable which 
changed after pariialling from positive to negative. This stage of the data 
reduction can be considered a crude form of criterion scaling. 

Process Sefr I became the sum of the following clusters: ratings of 
teachexs, opportunity, traveling frequency I, autonomy II, and percentage 
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of unique assignments. Process Set 2 became the sum of the following 
clusters: autonomy I, traveling frequency II, and the number of days since 
the last test. Process Set 3 is the single variable, the number of years of 
teaching experience with the program. The last variable i^ interesting be- 
cause while teaching experience is correlated with outco "^ , it is r ore 
highly correlated with input, iuaicatin.b tl at more expcriencea teachers 
ate working in classrooms with more gifted studeaits; but judging from the 
partial correlation results, they are not any more successful in teaching. 

Data reduction; Future . In the future, it hoped that the data will 
be collected in a way which will permit more logical and parsimonious 
reduction procedures. However, there will still be several steps involved. 

1. The data should be checked to eliminate variables with little 
or no variability or measures with high numbers of obvious 
errors. 

2. Preliminary correlations should be examined to see how the 
variables relate to each other. 

3. Each variable should be examined to see if a tranf ormation 
is necessary in order to make the variable more additive 
with respect to the dependent "variables. The exploratory 
data analytic techniques pioneered by Tukey (1970) and 
others are particularly relevant to these problems. 

4. Logical clusters of the process variables should then be 
formed with the aid of factor analytic techniques when 
appropriate. 

5. Preliminary correlations and partial correlations should 
be run on the reduced data set. 

In summary, a particularly difficult problem in arry research effort 
involving large numbers af variables and limited numbe:r:s of observations 
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is the reduction of data* Over the past three years, we have evolved a 
system of exploratory dat-^ analysis to help in da! 'eduction which keeps 
as much of the origi: ' anon as possible, ^ educii the abso- 

lute number of variables \v nne is dealings It has been out experi- 

ence, hov/ever, that practitioners in the field are still most interested in 
the relationship between specific practices and student achievement* There 
fore, it is important to have specific information easily retrievable and to 
display the data reduction process. 

Analysis 

This section considers how the impact of classroom processes on 
student achievement has changed over the three-year period, how this 
problem should be viewed, and the directions for future research. The 
basic question is; What is the influence, if any, of the educational process 
on student achievement? One of the more convincing methodologies used 
in recent studies in this area is commonality analysis (see, e.g., Mayeske, 
1970; Mood, 1971; Cooley & Emrick, Note 3). Commonality analysis is 
an extension of multiple regression in which the total variance explained 
by the model is partitioned into variance which is unique to each predictor 
in the model and variance which is common to the group of predictors. 

Results . Tables 5 and 6 present the results of the correlations and 
partial correlations of the reduced data set. From these tables, we can 
see that it is easier to determine what not to do in a classroom (Set 2) than 
what to do in a classroom (Set 1). We can also see that this procedure of 
reductions has substantially redxKZ^ed tke traditional problem of collinearity 
among the data. 
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Table 5 

Correlations Among Sets for 1973-74 Data 





Input , 


. ,^ Set 1 


Set 2 


Set 3 


Outcomes 


Input 


1.00 










Set 1 


.15 


1.00 








Set 2 


-.14 


-.31 


1.00 






Set 3 


.61 


-.04 


.05 


1.00 




Outcomes 


.91 


.29 


-.33 


.49 


1.00 



Notejrn = 55. 



Table 6 

Partial Correlations Among Sets for 1973-74 Data 







Set 1 


Set 2 


Set 3 


Outcomes 


Set 1 


1.00 








Set 2 


-.30 


1.00 






Set 3 


-.17 


.17 


1.00 




Outcomes 


.38 


-.50 


-.20 


1.00 


Note: n = 55. 
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If wc analyze this reduced data set using commonality, the results 
indicate that none of the process data taken separately is significant in 
oxplainin^ achievement. However, the results of an analysis in which 
all of the process data are combined and considered as a unit are more 
promising. Table 7 shows that 88% of the variance in end-of -year achieve- 
tnonl is accounted for by input and process information combined. Input 
alone explains a total of 83% of the variance, 47% of which is unique to 
input, while 36% is common with process. Process alone explains a total 
of 4 1% of the variance, 5% of which is unique to process. 



Table 7 
Commonality Analysis 






Input 


Process 


Total 


.83 


.41 


Unique 


.47 


.05 


Common 


.36 


.36 



Note: = .88; n ^ 55. 



In comparing this to previous years, it is most important to consider 
thr 7,oro order correlations between student input and outcome. These cor- 
rt-lationrt have steadily increased since 1972 from .60 to .80 to . 91 as a 
iTHuli of ihc increased precision with which entering and end-of-year abili- 
tirrt are assessed. The total has also increased from . 68 to . 72 to . 88. 
'Hio variance which can be uniquely attributed to classroom process has 
K'onc? from a high of 14% in 1972 to 7% in 1973 and 5% in 1974. Undoubtedly, 
thifi liaB boon adversely affected by the increasing size of the relationship 
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between input and outcomes and by the limitation of using one educational 
program. This can also be explained by the fact that as classroom proc- 
esses become more and more stable over time across grade levels, the 
power of entering student performance to explain end-of-year-p<?rf ormance 
is likely to increase. 

This analysis has several implications. First, initial student abili- 
ties consistently dominate the explanation of end-of-year student achieve- 
ment* Second, classroom processes are contributing a small but consis- 
tent amount to the explanation of student achievement. Third, we are not 
yet ready to use these techniques to help identify the contribution of specific 
claijsroom processes which affect student achievement, not only because of 
the limitation of sample size, but also because of the examination of only 
^ne educational program. Currently, the best we can do is to point to prac- 
tices which are consistent in their relationship to student achievement. 
Precision of assignment procedures, tutorial practices, and appropriate 
use of opportunities seem to improve student achievement, while negative 
and erroneous tutoring reduces student achievement. 

Discussion 

In this paper, I have argued that public policy evaluation requires that 
one include information about the nature of the program or the innovation 
rather than engaging in simple black box contrasts. In the process of meas- 
uring implementation, large amounts of intercorrelated variables are gen- 
erated. In order to use the data in further analyses, they must be sub- 
stantially reduced. Approaches to data reduction are, therefore, outlined. 
Finally, by using a form of regression analysis, outcomes are analyzed by 
taking into consideration not only initial input and programmatic descrip- 
tors, but also the interrelationship between the two. It is a fact of social 
innovations that the innovation is confounded with consumer self-selection, 
not only in education, but also in health care and other areas. Stated 
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differently, social reforms are rarely randomly distributed; therefore, 
techniques which take into account and describe the nature of the confound- 
ing must be used for evaluative analyses. 

This paper has used an empirical study of a particular educational 
innovation to demonstrate these points. In order to improve this particu- 
lar line of research, several steps must be taken in the future. The data 
n,ust be gathered according to a theory both to improve the interpretability 
of the results and to make the requisite data reduction more logical. The 
lumber of classrooms on which data are collected should be increased. 
And finally, the process information should be gathered from several pro- 
grams instead of just one. 
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APPENDIX A 
Description of the Instructional Setting 
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APPE^mDIX A 



D . ♦ . V itzTit m of the Instructional Set erg 

The basic c harac^te^ristics of the LRDC instructional model are that 
(a) it provides an eiiv7fera?i"^^ent that is adaptive to the educational needs of 
the student, (b) tbi* zMrv\z ;la are crrg?snized and presiented in a way tiiar 
attempts to teach a' i rg^imforce basic cognitive skills, and (c) the studemt 
directs and contr: \ v.s t \ her own learning within die context of the cur- 
riculum. Operati.-^^ - :-nis means that specific subject-matter areas are 
broken down into t v ^ ^^^^ which are hierarchically sequenced 

in a curriculum. acrenG:*:ent tests, cur riculum- embedded tests (CETs), 
and posttests are c .fsstrracted for each unit so that students can be initially 
placed in the curricL li=x.a.nd so that their success in learning the material 
can be continuously nnoiiitored. Using the information provided by these 
tests, prescriptions rwr itten for each student on a daily, weekly, or 
monthly basis. If 3. sziadent does not pass a GET, (s)he continues working 
on the same objective inr^unit, with additional practice work. If a student 
does not pass a posinr^sEsrt, i^s)he is recycled through the appropriate sections 
of the. curriculum amr retested. 

The class rooxn.ai2ix=Eally has cone teacher and one assistant or aide 
for 25 children. llssssHiy;, the teacher and assistant both circulate (travel) 
around the room during the work period; sometimes one will circulate 
while the other administers or corrects tests or tutors small groxips. 
While circulating, tlxe teacher or assistant corrects the work being done, 
occasionally alters 3r=«nxiptions„ offers brief tutorial assistance, and 
supports the studenit ezmotrcinally and academically. In general, the morning 
is divided into a work peadiod in which assignments are prescribed and an 
exploratory period during which the children investigate cur riculum- related 
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Appemdix A (Cont'd) 



mater la^l on their own. Classrooms vary in the degree t07%r::riicii srzrdeiats 
control the isjr:ision points Lr. their daily learning situaricn, such a^s: which 
subject to s^ady and when, which assignment to do first, wtr-an and w3aen not 
to work on formal curriculum, when to take a posttest, wi^n to cha^rge units, 
and so on. Thus, while the currieulum is relatively con^rr:: ent f rona class- 
room to cla:i5sroom, there axe many areas in which there; :ci:2 be diEfexences 
in implementing the program. 
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APPENDIX B 



Deiscripiion of Process Vaoriables ... 31 

Qoxestionmaire 

Videotape Coding Scheme 5.3 

Videotape Rating Scheme 5:6 
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APPHr^IX B 



Description of ' ocess Variables 

IntLiTrumemrs 

Thir-rr- irin- -e of the variaile^^ rvere obtarned from the 15-item ques- 
tionisaire T:^*h: ':ii-vras administeredi t^ice during the year. The local educa- 
tiona-l sp^r ^"-' r-pmpleted the forms in November and March, and the data 
frorrx both 3:drrdmst rations were tfc-n averaged. 

Four '^^^ri^bles v;'ere obrainei^^ fr-om the sirudent profile sheet which 
was cumuladiive s:rit& prx^vided accuxaite and easily accessible information 
about studCTl placememt in and progress through the math and reading cur- 
ricula. These shstets were kept by the class rcrom teacher or aide and were 
sent to LEIDC forir t±mes du:ring the school year. 

One '^■3:ri^ble, time estimate , was obtained by using school records 
to deter^mbie the numfb.er or days La the academic year, questionnaire infor- 
mation cr numbrer of mirnutes in trie school day, and percentage of children 
presenizm the s^mplt^ day ci^: estimator of attendance. These three 
number:-5 were then Tr::zltiplit-^^ ^ yield an estimate of nun.yier of minutes 
ST>ent t?' schoDCii durinxr -the ye:^ . 

JZ^trfcr-vari^ables wetre ohut«jdned from videotapes recoxded for 15 
^iirate- izrr ::2ssding ^and 15 x=dnn±es in math in the fall aTid again in the 
spff :ng i^drtTLsn siiTrming; ^o^^r th^ two collections. The tapes were made 
by local ^"^^leriulists at thf^ sKt^es while the class was in progrrsss. Codings 
were adioiistEd for variatmna in time so that all data would be based on a 
total crt:>«n oE vidaercrnape-per classroom. 

TTkermEestionnaire and videotapes focused on gathering information 
from sevfin -major domains. These domains will be described briefly and 
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Appendix. B (Conir-jd) 

the data suinmari:::ed. It sh juld be m:£iitioned ihat the assignment of varia- 
bles to domains is to some cjxtenr 2.rrdtrary, since one "variable may, in 
reality, belong to several domairu . Thus, the sets should not be interpre:tted 
as rigid definitions of the domain- aiij£igning varic^bles to domains is a con- 
venient way of organizing and corLiiider±ng a rather large number of class- 
room process variables. Tar le 5-1 lazovrs the doimaLns ari;V variables. 
Tables B-2 through B-8 pres^^nt he r."e:::5ult3 by- sitre. 

Domains acid l/ariables 

ClasrsTQom context . This daniain consists af five variables: the 
teacher's csral years of teaching es^^erience, the teacher's years of experi- 
ence in working with the LRD Z pro-am, the tnfcal number of children en- 
rolled in the cla-ss, the occur^rrencs: cof an erxtemta.! event ^wmich had a severe 
effect (crisis) on the classroom, szid the availability of:.a trained alternate 
in case of teacher absence. Althoixiigh these variables do not relate directly 
to the LRDC program, they are likalEy to affect the resaiil-ts of thai:: pxogr-am. 

Table prc^^^nts eHkv site averages for these five variables. The 
sites vary tremi.rnd,ou^ly wit: : r n^gar£ to some of these variabjies. For ex- 
ample, teacher ex^ierience : --.^es from an average of 1« 2 yeaxs Site DO 
to 18. 5 years at Sites 1 and ... 17ne se;cond variabie, tea^oher experiesnce 
with the progxsxsi, is det^: jLencerrrU tiEnie number of ;grears ttrat UEID^C has bee 3i 
involved with .eaich sixe;., Thp thiird ^rariable, the mimber nif childiren enrorii^d, 
shows less vsnrfribflity iranir. the pa^t. The 1972-73 rangre of sifce means 
was 20.6 to 2T. 0; ihe pn::essnt daxa ranges from 20. 1 to 25, 8, No doubt tins 
is a reflection 2^f reduced enrollment. Such a reduction nrray affect the im- 
plementation of anindividualized program in that the quality and quantity til 
time a teacher is a:ble to spend with each student may be sSected* QnLy 
two sites repcrrte^ any crisis sitostion which may have hasii ai: effect om rail, 
or some of th^ $xtudcnts. The last v^ariabie in Table whr!±hcr or Tiot3. 

trained alterrrEatirj was available in r^se of teacher absenc^i, . shows that abtacx: 
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Table S-1 

Classroom Process Variabjes 



Classroom context 

Total teacher experience in number of y ear 

Total tether experience %vjth the LRDC instructional model in - lurrtir^r of /ears 
Numtx^ of children enrolled in the class 

Whether or not some external event had a detrimental effect on ■:n2 r-^ssroom (e.g., flood, epidemic, etc) 
VVhether or rnot a trained alternate was svaiilisble in case of teacher om^nce 

Allocation of tf^me 

Percentage of time spent iin individu3liz.ed arnnri small group activities 

Percentage of time spent in large group :aa::wittes 

Percentage of time spent in free-choice aciiiMhties 

Number of minutes per day in IPI readfrrg 

Number of minutes per day in IPI math land nrath cirnill 

Number of minutes per day spent watcbtn^ eEoicatEOTial T\f 

Time estimate-average nutmber of minutes uanrt in school during The academic year 

Assignment procedures and monitoring stuoent p. '^qpgss 

Percentage of unique assignment-the peTsjienH^c uf th^r totall ass!Grrr-t«^»^ts5:gtven whiich weiEf Siifferent from 

any ether assignment given 
Percentage of blocked assignments-where s. blcaked assignment ws^ :r:ir^;or morei cssnset:ujtiv5=rn32iis or 

boxes with less than an equal number ofi' iscolEted pages or boxer 
Whether or not student's work is checked wMq traveling 
Wumber of days since the last posttest was g^ryi^n 
Percentage of students passing the lasrt postttii^t 
fu'umber of days since the last test of any tv^pe -•«'Jtc;c:i'»} ii 
initial student placement in 'hennath cur^r::li;'u^' 
Student progress in the mattr; currriculiaTn 
Initial student placement m "he readinci curricu. ^im 
Student progress in the treadiing curricuium 

Provisions for student self^direccron 

Children decide what unnit ofrmath to study 

Children request special tutoring 

Children begin work as soonas they arrive au:sCTiDl 

Peer tutoring occurs 

Children get their own assignnnenis 

Children get their own prescjiiption tickets 

Children decide when to take a test 

Children move back and forth betwer^n wcm^ and free actiwrtw 
Attendance 

Percentage of children present on sanrrpnr-tiijays 

Management procedures 

Number of adults who travel during msath 
Number of adults who travel durirDg raading 

continued 
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Table B-1 



Frequency' c.f instructional event (Ail of the information below is available for both math and reading.) 

Total number of interactions by the teacher and a child during 60 minutes-contacts 
Number of times a teacher goes to a child but makes no verbal comment-checkoffs 

Number of times the teacher makes a management statement or question (i.e., non-cognitive, no information 
is passed) -TMSO 

Numberof times the teacher makes a cognitive statement which requires no response from the student-TCS 
Nurjiber of 5imes the teacher makes a cognitive question (any statement which requires an overt response 
ftrom line 5tudent)-TCQ 

Percentage of times the teacher makes a statement or question which is negative or punitive in nature 
Number of times the teacher spends more than one minute with any one student-tutor 
Number otf times the teacher makes a cognitive error which is not corrected 
Number oi times a student initiates an interaction mih the teacher-child contacts 

Ratincgs of in^struc tional events ( 1 -5; 5 is high) 

Active responses are sought by the teacher 

Th?9 teacher uses backward chaining 

Th^e teacfnsr models student responses 

Tbe teacbtir refers to earlier curricular information 

The teadi^er focuses the attention of the students on task 

The teachier solicits the opinion of students as to correctness of their responses 

The teacher refers to earlier success of students 

The teacnef solicits the attitude of students toward the curriculum 

The teap^ar uses praise which is contingent on student work 

The teacner uses general praise of student behavior 

The teacher's tutorials are clear, correct, and understandable 

Tme teacher is able to manage the class as a whole 
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half of the second-grade classrooms do have an available and trained sub- 
stitute# 

Allocation of time . The data gathered in 1971-72 indicated that there 
was a positive and significant relationship between the amount of time spent 
in mathematics per day and the level of mathematics achievement (Lein- 
hardt, 1974a). During 1972-73, more extensive information was gathered 
on the way in which individual teachers and sites spent the time available 
to them (Leinhardt, 1974b). The focus of this domain was centered in 
three areasj First, , what percentage of the day is spent In specific modes 
of instruction- -individualized and small group activities, large group activi- 
ties, and free-choice activities? Second, what amount of time per day was 
a child exposed to specific subjects such as reading, math, or educational 
TV? Third, a general time estimate of total minutes spent in school. 

The first three variables in Table B-3 deal with the percentage of 
time spent in individual, large group, and free-choice activities. We 
expect that at least half of the school day is spent in individualized activity, 
since the majority of LRDC's instructional program is individualized. 
From Table B-3, we note that seven of the ten sites do, in fact, spend the 
better part of the day in individual and small group activities. When the 
means of Variables 1 and 3 are combined, we find that all sites spend more 
time engaging in activities classified as individualized than they do in large 
group activities, although there are considerable differences across the 
sites* This is an encouraging sign since this information refers to the 
entire school day, not just the 3 1/2 hours during which the program is 
implemented. The amount of time spent in mathematics has increased 
for all but Site 9 since last year, whereas the amount of time spent in 
reading has decreased for all but Site 4. We also collected data on the 
amount of time spent watching educational television to determine how 
many of the sites use this media as a regular part of the academic day. 
It appears that the classrooms that do watch educational TV do so on a 
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Appendix B (Cont'd) 

small scale and, therefore, little effect on achievement is expected. More 
sites are using educational television, however, and for longer periods of 
time than last year, with the most significant increase at Site 3* The last 
variable reported in Table B-3 is an estimate of the average number of 
minutes a child spends in school during the year. From the range of means 
it can be seen that a child may spend as little as 48, 000 to as much as 
70,000 minutes in school. 

Assignment and monitoring procedures . This domain deals with the 
assignment and monitoring procedures used in a classroom. It is through 
the accurate matching of student needs and curriculum content that much of 
individualization is achieved. The first variable in this domain is the per- 
centage of unique assignments, that is, the percentage of assignments given 
which are different from any other assignment given with respect to level 
and skill. The second variable is the percentage of blocked assignments 
which is defined as three or more consecutive tasks or pages assigned with 
less than an equal number of isolated tasks or pages. These variables tap 
the way in which the teacher makes assignments in class by using mathe- 
matics assignments as the sample of assignment practices in general. It 
appears that the teachers in all of the sites make unique assignments. The 
lower limit of the range of means across sites has been raised to 70. 3% 
over the previous year's 57% (Leinhardt, 1974b). All but two sites have 
increased the percentage of unique assignments, and those two had minimal 
decreases. The percentage of assignments which are blocked decreased by 
about 12 percentage points across sites. The third variable relates to the 
kind of feedback a student gets on progress, that is, is his/her work 
checked during class while (s)he is at work. Table B-4 shows that at all 
but one site a student's work is checked during class by the teacher or 
aide who travels. 

The LRDC instructional model also provides for frequent systematic 
assessment of the students' learning progress; that is, it provides procedure 
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for the teacher to monitor the movement of students as they work. The 
assessments are generally in the form of individual, self -administered, 
criterion-referenced tests that are of three main types: pretests, 
curriculum- embedded tests (CETs), and posttests. It is expected that 
children will take at least one test of some type at least once every five 
to seven days and a posttest once every twenty days. The number of days 
since the last posttest varies considerably across sites. ThB averages 
range from 8.5 to 23.2. Without these two extremes, the range is be- 
tween 14.6 and 18.0. The percentage of students passing the posttest is 
high across sites (84.4% to 97%). The next variable is number of days 
since the last test of any kind. The range is from 2. 3 to 6. 8, with the 
highest concentration of averages around 2. 5, The last four variables in 
this domain deal with where the student started work in the math and read- 
ing curricula (placement) and the amount of work the student completed in 
those curricula (progress). It is evident from the site averages that there 
is a large amount of variation among sites in the placement of students in 
the math curriculum (3.6 to 25.5) and slightly less variation in reading 
placement (3.0 to 14.0). Both variations reflect basic differences in enter- 
ing ability among the students at different sites. Progress within the math 
and reading curricula also varies (6.8 to 13. 1 and 6.0 to 10.0), but there 
is not nearly as much variation as in initial placement. 

Provisions for student autonomy (self-direction) . One of the continu- 
ing long-range goals of the LRDC model is to develop independent learners. 
In accordance with this goal, there are many ways in which a teacher can 
provide opportunities within the classroom for the student to learn to exer- 
cise autonomy. Table B-5 presents the site averages on eight autonomy 
measures. 

Attendance . Student attendance is an important variable in the imple- 
mentation of the LRDC program. Generally, if the students are not present 
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Appendix B (Cont'd) 



in school, they cannot benefit from instruction. Although an individual 
program makes student absenteeism less damaging than a traditional pro- 
gram, it is still an important variable to take into account. Good attend- 
ance can also be taken as an indication of a positive attitude toward school 
in general. Table B-6 reports student attendance based on the percent of 
children present on two randomly selected days. The means in general 
are high and comparable to last year's data. ' 

Management procedures . The number of adults who travel during 
mathematics and reading is reported in Table B-7. It appears that most 
classrooms have two adults traveling during these classes. This is contrary 
to expectations, and it is not clear that this is the most efficient use of 
teacher time. 

Instructional event . This domain deals with ways in which teachers 
may differ in in-class behaviors. Our concern here is with the basic inter- 
actions between a teacher and a student during a 60-minute period. The 
variables included relate to: the rate at which the teacher moves from 
child to child, the type of contact (s)he makes with the child, the length of 
time (s)he spends with any one child, the effect of the interaction (i. e. , 
positive or negative), the number of cognitive errors the teacher makes, 
how often the students initiate an interaction with the teacher, and a list of 
teacher characteristics on which (s)he is rated. Teachers vary along all 
of these dimensions, and an individualized program may require, for exam- 
ple, a faster rate of movement and greater frequency of contacts than a 
traditional one. Through videotaping classes in progress, we hoped to be 
able to take a closer look at the variables deemed significant in the domain 
of the instructional event. 

Table B-8 presents the 30 variables derived from videotaping teacher- 
child interactions in each second-grade classroom for 30 minutes of read- 
ing and 30 minutes of mathematics. The total number of contacts in reading 
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and in mathematics shows that there is a great deal of difference across 
and within sites in the rate at which_the teacher travels. The respective 
ranges of means are irom 33.2 to 80 and 29. 8 to 82, with overall means 
of 45. 38 and 50. 11. 

The next four varia&les in reading and math deal with the types of 
contact the teacher has with the child. The teachers appear to make mast 
of their contacts in management areas and in asking cognitive questions^ 
Teachers appear to make_few cognitive statements in reading. In mathe- 
matics, the number of cognitive statements is greater, but means are less 
than those for cognitive questions with the exception of Site 1. 

The percentage of negative statements is small in both reading and 
math. The tutors variable is characterized as any interaction between 
the teacher and a child branch lasts longer than one minute. Here again, 
the frequency is relativelry small, with mean reading tutors ranging from 
.5 to 8. 6 and math tutors ranging from 3. 0 to 8. 2. The interpretation of 
this variable is ambiguous at this time. ITiat is, do tutorial interactions 
increase the quality of instruction with a given child to the extent that it 
overcomes the diminished rate of travel and frequency of contacts? Or, 
is it more valuable to interact with more children, even if for a limited 
time? The number of cognitive errors made by the teaclier in reading and 
math are very minimal (0 - 1.2). 

The final variable directly related to reading and math was the num- 
ber of times the child initiated the interaction with the teacher by either 
making a statement or asking a question. Once again there is substantial 
variation across sites (means of 3. 3 - 22. 0 in reading and 4, 7 ^ 19- 0 in 
math), as well as within sites (standard deviations of 3. 2 - 22.6 in reading 
and 1.6- 12. 7 in math). 

n^e last 12 variables were scored on a rating schedule from 1 to 5 
based on observations of the classroom videotapes, with 5 meaning the 

50 



46 



Appendix B (Cont'd) 



teacher used the xiamed skill often and well. Teachers in all sites appear 
to seek active responses from the child, since the lowest m^san is 2. 5 (only- 
one site) and the others range from 3.Z to 4. 3. On the othe~hand, it would 
appear that there is very little backward chaining in instruczxon, since all 
means are between 1. 0 and 1.6. ^'Modeling responses*' seems to be a more 
frequent form of instruction, but is stilLinot extensive acros isites, and 
referring to earlier curricular information occurs only sligniiiy mone than 
backward chaining. 

Table B-8 also indicates that teachers across sites doz-focus the atten- 
tion of students on task. This may be related to seeking active responses, 
since the means for both variables are quite similar and of the same mag- 
nitude. The next three variables seem to occur only occasionally with 
smaller variation across sites. Two of these variables relate to solicit- 
ing feedback from the child on specific work done and on the c^irriculum in 
general. Contingent praise refers to positive reinforcement for specific 
Work completed, whereas general praise is not related ±o any specific cog- 
nitive task. Across sites, contingent prai^ is used moTe often than gen- 
eral praise and appears to be ^sedfrequertfiy': 

The next two variables r^ate to tutoxing and management behaviors. 
The tutoring rating was based on how well die teacher tutors a child: Is it 
clear, concise, and understandable by the child? Management: was rated 
according to how well the class as :a whole functicms:: Do the ciildxen know 
what to do and do it, and are the class and the tocher moving efficiently? 
The program seems to be working well on both counts, since the ratings in 
these two areas are fairly high with the exception of Site 10 in management- 
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Questionnaire 



ERIC 



Name of Individual 
Filling Out Questionnaire 

Date 

District 

School 

Teacher's Name 



1. Hojv many years of teaching experi- 
ence (prior to this school year) does 
the teacher have? 

2. H<iw ^v^eiy year s has the teacher 
been Using PEP or IPX prior to this 
school year? 

3. How many children are enrolled in 
this classroom? 

4. How many children are present 
today in this room? 

5. How many adults are normally in 
the room? 

6. How many adults normally travel 
at kny one time? 

During math: 
During reading: 

7. Does the teacher (or aide) check the 
majority of the children's IPI/PEP 
work while traveling ? 

For math work: 

For reading work: 



If either of the above is no, when is the majority of the work checked? 
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DATE' THIS QUESTION IS BEING FILLED OUT 

8. Please list each of today's IPI/PEP math and/or quantification assign- 
ments for each student. Please also list the DATE and SCORE of the 
last POSTTEST given. Record the posttest score regardless of whether 
or not it was passed. (Note ; If assignments are made by the week, 
record the entire assignment. ) 



Please record the date and score of 



Kxam- 
plr: 



Unit 



Level 



Skill 



Pa^ e s 



the last POSTTEST given. 



• Add /Sub 



1-4, 7 
1,8, 10 



POSTTEST 3/18/74 74% 



Reminder ; Did you record all of the 



dates and scores? 
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9, Check those activities which you observe occurring regularly in this 
room. 

if 

Students obtain own assignments for math (as opposed 

to having them passed out), 

Students obtain own notebooks for reading. 



Check 



one 



Students sometimes write their own prescriptions 
(other than exploratory) for math, reading, or 
science. 

Students are consulted as to the mode of instruction 
(tapes, books, texts, or games). 

Students select time of day for work in subject-matter 
area. 

Students help to select which unit of math (geometry, 
money, fractions) will be studied. 

Students suggest when to take GET (either by request- 
ing or delaying it). 

Students suggest when to take Unit test. 

/students move back and forth between exploratory 
I and prescription areas. 



, Students go to exploratory any time after completion 
7 1 predefined list of assignments. 



Teacher decides when exploratory period begins for 
V the majority of students. 

Students request or bring in activities for exploratory 
areas. 

Students request small group lesson. 
Students request special tutoring. 
Scudent's opinion of curriculum or task is sought. 



Give an example: 



Students sometimes choose to work with other stu- 
dents rather than alone. 

Students may start work as soon as they come in 
in the morning (or after lunch) rather than waiting 
for the teacher to start the period. 
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Peer tutoring occurs, 

Teacher seeks information about the attitudes of 

students toward their work. 

10. Please give an approximate number of hours and minutes which any one 
student would spend during an entire school day in each of the following 
instructional situations. The total number of hours and minutes should 
add up to approximately one whole day , not just the IPI part of the day. 

Student works alone with occasional adult or peer 

tutoring. 

Student works in a small group with adult instruc- 
tion (2-8 children). 

Student works in a large group with adult instruc- 
tion (9-whole class). 

Student is engaged in free-choice activity, 
Other. Specify 

11. , Please list the number of minutes per day students in the classroom 

spend in the following activities: 

IPI Reading 

IPI Math 

Some form of math drill 

Wathing educational TV 

( Note ; If educational TV is unavailable in your 
school due to either lack of reception or availa- 
bility of TV monitors, please indicate above. ) 

12. Is a math maintenance program being used in this 
room (systematic drill of number or algorithm 
facts three or more times a week for 10 minutes 

or more)? . 

13. Any unusual events which in your opinion would strongly affect the 
nature of this classroom (such as floods, severe illnesses, etc.)? 



14. Does this classroom have trained, pei^manent alternates 
who substitute in the case of teacher or aide absence? 



51 



Appendix B (Cont'd) 



15. Observe the teacher for 10 minutes during the math or reading period 
and record the number of negative comments made by her or him. (A 
negative comment is one that generally orders a child to stop some 
activity or is sarcastic or punitive toward a child or group of children. 
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Videotape Coding Scheme 

Videotapes are observed using the teacher-student interaction as the 
main unit of coding. Every time the teacher changes from one child to 
another, that interaction is recorded according to what the teacher says and 
the response(s) of the child or children. There are eleven categories that 
may be checked for each interaction. The following is a summary of the 
guidelines used to determine if a category should be checked. 

1. Checkoff (J) 

The teacher checks a child's work and leaves without making any 
comment to the child, i. e. , there is no verbal communication. 

2. Teacher Management Statements or Questions (TMSQ) 

The teacher says something to the child (or children) which con- 
tains no cognitive information but serves a managerial function only. 
Management statements or questions may concern curricular materi- 
als, discipline, personal functions, or information about placement 
within a curriculum. General examples are: ''Yes, you may go to the 
bathroom"; "Get your materials ready"; "Has the aide checked your 
work?"; "Be quiet; sit down"; "Go to page 10"; "Good" or "Right" (if 
"good" and "right" occur by^ themselves , they are considered manage- 
ment information); etc. 

3. Teacher Cognitive Statements (TCS) 

The teacher makes a statement containing cognitive information 
which does not elicit a response from the child or which elicits a 
response which is an echo of the teacher's statement. Cognitive state- 
ments may occur when the teacher reads directions; the teacher asks 
a rhetorical question; the teacher asks a question and then answers it; 
the teacher gives the command, '^Repeat 2+2 = 4," and the child says 
"2 + 2 = 4"; or the teacher lectures. 

4. Teacher Cognitive Questions (TCQ) 

The teacher asks a question about cognitive material which elicit s 
a response from the child. Examples of teacher cognitive questions 
are: "How much is 2 + 2?"; "Read me the sentence that tells what Sam 
is doing"; "Read the directions which tell you what to do on the nexfc 
page"; and "Read this sentence. " It can also be the kind of statement 
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which appears to be managerial but which clearly contains cognitive 
information. Consider the following set of questions: 

''How many pages did you do today? How many were you supposed to 
do? How many are you going to do tomorrow? Did you do more or less 
than you had planned to do?" Questions that give detailed discussion 
about the titles of hooks, how to spell them, and how to map them out 
are also considered to be cognitive. Although the above examples are 
teaching children skills not normally taught in the curriculum, they are 
clearly cognitively based skills and are, therefore, counted as cogni- 
tive questions or statements. If a cognitive question occurs with a cog- 
nitive statement, only the cognitive question is counted in the summary 
of the videotape data. 

5. Child Cognitive Statements (CCS) 

The child makes a statement regarding cognitive information in 
response to a question, without a prompt, or reads some cognitive 
material- -a story, directions, and so on. (This category is only coded 
in the presence of a teacher because the videotape camera follows the 
teacher* ) 

6. Child Cognitive Questions (CCQ) 

The child asks a question which elicits a response from the teacher. 
This may be in the form of a statement or a question. For example, 
''How do I find this answer?" or "I don't know how to do this. " 

7. Child Management Questions (CMQ) 

The child asks a question or makes a statement which elicits a 
teacher response dealing with a management area. Examples are: 
"Where Ls the tape for this lesson? "I don't have a pencil"; "May I 
sit next to Susan?"; etc. 

8. Child Management Statements (CMS) 

The child makes a statement requiring no response from the teacher 
in a management area. Examples are: "I fell on my way to school"; 
"Johnny hit me"; "The aide checked my paper"; etc. The rules of state- 
ments versus questions are not semantic; they are based on whether d'r 
not a response was made. (The same rules apply with Teacher State-^. ^ 
ments. ) 

9. Tutor 

An interaction is counted a tutor if it lasts over one minute. 
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10. Error 



The teacher makes an error and does not correct it; for example, 
"2 + 2 = 5. This is not a pedagogical error; this is an error of infor- 
mation. 



1 1. Negative 

Any category in which the teacher's statement is essentially puni- 
tive is considered to be negative. The statement can be considered 
negative when the tonal quality of what is said is punitive, not just the 
content of the statement. '^You did very well on that work, if said in 
a sarcastic and degrading manner, may mean that the teacher is telling 
the child how terrible (s)he is doing. Most negative statements involve 
demanding the stopping of behaviors on the part of the child. Examples 
of this are; "Stop it»'; »»Sit down"; "You weren't listening". "No" or 
*'not" might be the cue. Repetition such as, "I've told you that three 
or four times, " or a negative facet attributed to the child's overall per- 
formance, as " You never listen when you are being told something, " 
are also cues to negative statements. 
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Videotape Rating Scheme 

At the end of observing the videotape of a teacher, a rating scale is 
filled out. There are 12 rating categories. Ratings a.re used instead of 
frequency counts because frequency coding did not turn cut to provide addi- 
tional information in these areas, and it was easier to rate than to count 
frequencies for these categories. The coders are instructed to know the 12 
categories before viewing the tape. They are not to fill out the rating until 
they have watched the whole tape, but they may take notes. Each category 
is rated on a scale from 1, indicating never, to 5, indicating frequently. 

1. Active Responses Sought 

The teacher elicits active responses from the students. That is, 
students are required to verbalize, manipulate, write, or perform 
some active form of behavior when the teacher interacts with them, 
as opposed to passive behavior such as listening, 

2. Teacher Backward Chains an Explanation 

The teacher, while tutoring, gives the child a complete explana- 
tion and elicits behaviors from the child in a backward chain. The 
teacher uses the last stimulus of a chain as a reinforcer for the behav- 
iors required in the next to the last step, and so on. 

3. Teacher Models a Response 

The teacher very clearly shows the child what a response consists 
of. For example, the teacher says, "3 + 4 = 7. Write the 7 in the box. 
(S)he then shows the child how to do it and watches as the child does 
the next problem. . 

4. Teacher Refers to Earlier Curricular Information 

The teacher mentions, during an interaction with a student, infor- 
mation that was previously learned as in, for example, ''You sounded 
out v^ords like this before. It is a communication to the child that the 
task the child is facing is something not totally new. 
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5. Teacher Focuses Child's Attention on Task 

This may be judged not only by what the teacher says or does, but 
also by what the child is doing. That is, does the child look off into 
space or at the materials? Does the teacher let the student wander off 
on a tangent, or does (s)he keep the student on task? 

6. Teacher Solicits Child's Opinion on C o rrectness of Response 

The teacher asks the child to determine whether his/her response 
is correct. 

7. Teacher Refers to Earlier Success 

The teachrir mentions something the child has done well in the 
past, as, for example: *'You really did well with multiplication yester- 
day; division is the reverse process. See if you can figure out how to 
do these problems. You have done this kind of thing very well in th'^ 
past. " 

8. Teacher Solicits Information on Attitudes Toward the Curriculum 

The teacher asks the child how (s)he feels about the curricular 
materials or information. 

9. Teacher Uses Contingent Praise 

The teacher praises a student for work that is well done, for 
example, "You added these two numbers correctly; good," It may 
even be telling the child that (s)he consistently did something correctly 
This is contrasted with the next category, general praise. 

10. Teacher Uses General Praise 

The teacher praises a student in a nonspecific way: "You are 
working well, " "You have done good work, " or "You are in a great 
frame of mind today. " The child must interpret what is being praised. 
This does not mean that No. 9 and No. 10 are mutually exclusive. 
No. 9 is specific in relation to the behavior, and No. 10 is general. 

11. Tutoring 

This judgment is based on the clarity, conciseness, and accuracy 
of the teacher's tutorials. That is, is {s)he explaining things clearly, 
understandably, and logically? Is (s)hc to the point? Does (s)he answ- 
the child quickly and flexibly? Does the child seem to follow the expla; 
lion and use the materials appropriately? the student does not undei 
stand, docs the teacher take a new approach, become more concrete, 
or find some innovative way of expressing the thought? 
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12. Class Management 

This judgment is based on how well the classroom works, not neces- 
sarily on how low the noise level is. This category was developed be- 
cause it was clear that despite the fact that teachers had very different 
management procedures, some being quite constrained and strict and 
others being quite loose, within each type there were very different 
successes. That is, there are classrooms where the teachers are very 
structured and v/ant a quiet room but are not always telling their stu- 
dents to be quiet. Equally, there are classrooms that are open, free, 
and quite noisy and in which the teacher seems to be quite comfortable. 
What is basically being measured is the degree of harmony between 
what the teacher seems to want and what the children seem to te doing. 
Is the teacher fighting whatever is going on in the classroom? Do the 
children seem to know what is expected of them? Do they get materials 
easily? Is everyone functioning individually as well as in a unit? Is 
the process smooth and even, without major disruption? 
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